Verbal and nonverbal discourse planning

We are designing a Conversational Agent that communicates at the same time by speech and expressive face. As we are engaged in face-to-face interaction or in any communicative acts where the visual modality is available along with the acoustic one, multimodal signals are at work. Our communicative acts are performed not only through words, but also through intonation, body posture, hand gestures, gaze patterns, facial expressions and so on. Thus, an important step is to define how the Agent’s communicative acts are expressed into a coordinated, either sequential or simultaneous, verbal and nonverbal message. In making an Autonomous Agent capable of communicative and expressive behavior, then, a relevant problem to be taken into account is how the Agent plans not only what to communicate, but also by what (verbal or nonverbal) signals, in what combination and how synchronized. This depends on several factors : (i) consideration of the available modalities (e.g., face, gaze, voice), (ii) cognitive ease of production and processing of signals (for example; in describing an object, a gesture may be more expressive than a word); (iii) expressivity of each signal in communicating specific meanings (for example: emotions are better told by facial expression than by words); (iv) appropriateness of signals to social situations (for example: an insulting word may be less easily persecuted than a scornful gaze). Finally, metacommunicative constraints may occur that, say, lead to use redundancy when information to convey is particularly important or needs to be particularly clear: this brings to use verbal and nonverbal signals at the same time. Since few years, multimodal conversational systems [2, 3, 5, 18, 1] have been proposed. These systems integrate verbal and nonverbal signals such as deictic gestures, gaze behavior, communicative facial expressions. Cassell and Stone [4] have designed a multimodal manager whose role is to supervise the distribution of behaviors across the several channels (verbal, head, hand, face, body and gaze). Our system is close to this last one. But we are also considering the context in which the conversation takes place. In this paper we first describe our enriched discourse generator explaining the 2 sets of rules (trigger and regulation) we have added. We also review the different types of gaze communicative acts. Finally we present the variables defining the context and how they modify the computation of the display of the communicative acts.